{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all we need to install [ranx](https://github.com/AmenRa/ranx)\n",
    "\n",
    "Mind that the first time you run any ranx' functions they may take a while as they must be compiled first.\n",
    "Try running the code again to see actual usage speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install ranx"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download pre-computed runs from [ranxhub](https://amenra.github.io/ranxhub)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ranx import Qrels, Run, compare\n",
    "\n",
    "runs = [\n",
    "    Run.from_ranxhub(\"msmarco-passage/trec-dl-2020/ranxhub/ance\"),\n",
    "    Run.from_ranxhub(\"msmarco-passage/trec-dl-2020/ranxhub/bm25-pyserini\"),\n",
    "    Run.from_ranxhub(\"msmarco-passage/trec-dl-2020/ranxhub/distilbert-kd\"),\n",
    "    Run.from_ranxhub(\"msmarco-passage/trec-dl-2020/ranxhub/distilbert-kd-tasb\"),\n",
    "    Run.from_ranxhub(\"msmarco-passage/trec-dl-2020/ranxhub/unicoil\"),\n",
    "]\n",
    "\n",
    "# Import Qrels from ir-datasets\n",
    "qrels = Qrels.from_ir_datasets(\"msmarco-passage/trec-dl-2020/judged\")\n",
    "\n",
    "# \"-l2\" changes the relevance level for the binary metrics\n",
    "# as by instructions https://trec.nist.gov/data/deep2020.html\n",
    "compare(qrels, runs, [\"ndcg@10\", \"map@1000-l2\", \"recall@1000-l2\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Share pre-computed runs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build your run card\n",
    "\n",
    "A run card is a [YAML](https://en.wikipedia.org/wiki/YAML) file containing your run's metadata. The metadata that you add to the run card improve its discoverability and enable certain interactions on the hub, such as indexing and tag-based filtering.\n",
    "\n",
    "A run card is divided into four sections:\n",
    "\n",
    "- __Run metadata__: how the run was computed, who computed the run, tags for indexing, and other information;\n",
    "- __Benchmark metadata__: name, dataset, split, version, and the [ir-datasets](https://ir-datasets.com) id, if any;\n",
    "- __Paper metadata__: title, authors (with name and [ORCID](https://orcid.org/)), publication date, [doi](https://en.wikipedia.org/wiki/Digital_object_identifier), [dblp](https://dblp.org/) URL for [BibTex](https://en.wikipedia.org/wiki/BibTeX);\n",
    "- __Model metadata__: name, description, tags for indexing, paper title, authors (with name and [ORCID](https://orcid.org/))), publication date, [doi](https://en.wikipedia.org/wiki/Digital_object_identifier), [dblp](https://dblp.org/) URL for [BibTex](https://en.wikipedia.org/wiki/BibTeX).\n",
    "\n",
    "Here are an [empty run card](https://github.com/AmenRa/ranxhub/blob/main/files/runcard-empty.yaml) and a [filled one](https://github.com/AmenRa/ranxhub/blob/main/files/runcard-filled.yaml)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Share your run\n",
    "\n",
    "Once you have built the card, it's time to convert your run to the [ranxhub](https://amenra.github.io/ranxhub/share/) format and share it with the community.\n",
    "\n",
    "First, you load the run into [ranx](https://github.com/AmenRa/ranx).\n",
    "\n",
    "([ranx](https://github.com/AmenRa/ranx) support several ways of doing it.  Please, see [here](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/2_qrels_and_run.ipynb) for more information.)\n",
    "\n",
    "Once imported Qrels and Run, you can proceed with the conversion process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ranx import Run\n",
    "\n",
    "run = Run.from_file(\"path-to-run\")\n",
    "run.save(\"save/to/run.lz4\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to save your run in [LZ4](https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)) format to lower memory footprint. [ranx](https://github.com/AmenRa/ranx) automatically infers the file output kind, so you just need to add \".lz4\" at the end of the filename.\n",
    "\n",
    "We encourage you to submit runs accounting for 1000 results per query.\n",
    "\n",
    "Once finished, you are ready to share your run and its card with us by uploading them with this [form](https://forms.gle/fK6wLS83yZeoS1mL8). We will then upload your run on our servers and add the model card to this website.\n",
    "\n",
    "Pretty easy, right? :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ranx",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "692976b4e7dc85f192e7ff7f01132f0a9cfec5b2147f9c392e0014b728af08ff"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
